Skip to content

Gracefully handle errors in evals #2295

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 19 commits into from
Aug 29, 2025
Merged

Gracefully handle errors in evals #2295

merged 19 commits into from
Aug 29, 2025

Conversation

dmontagu
Copy link
Contributor

@dmontagu dmontagu commented Jul 24, 2025 •
edited
Loading

Also adds retry functionality. Builds on top of the work in #2282. Closes #2612

Still need to:

  • update the report printing to show errors in the terminal
  • make sure this change doesn't break the logfire display of pydantic-evals spans
  • probably add tests and docs

Note: I think this is technically a breaking change because it adds some fields to the ReportCase and EvaluationReport classes, so deserialization might not work on existing data. Given we are still 0.X I guess it's worth a bump to minor version(?) but I expect it won't be very disruptive in practice..

@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 24, 2025 •
edited
Loading

Docs Preview

commit: ae2ea45
Preview URL: https://a2326569-pydantic-ai-previews.pydantic.workers.dev

@dmontagu dmontagu force-pushed the dmontagu/graceful-evals-error-handling branch from 5fe2ebf to 6bb2a3f Compare July 24, 2025 04:20
Base automatically changed from dmontagu/retry-handling to main July 25, 2025 17:34
DouweM
DouweM requested changes Jul 25, 2025
View reviewed changes
@DouweM
Copy link
Collaborator

DouweM commented Jul 25, 2025

Note: I think this is technically a breaking change because it adds some fields to the ReportCase and EvaluationReport classes, so deserialization might not work on existing data. Given we are still 0.X I guess it's worth a bump to minor version(?) but I expect it won't be very disruptive in practice..

@dmontagu Can't we handle missing fields by just using the default value of []?

@DouweM DouweM self-assigned this Jul 25, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 2, 2025

This PR is stale, and will be closed in 3 days if no reply is received.

@github-actions github-actions bot added the Stale label Aug 2, 2025
@DouweM DouweM assigned dmontagu and unassigned DouweM Aug 4, 2025
@DouweM DouweM removed the Stale label Aug 4, 2025
@github-actions GitHub Actions
Copy link

This PR is stale, and will be closed in 3 days if no reply is received.

@github-actions github-actions bot added the Stale label Aug 16, 2025
@github-actions GitHub Actions
Copy link

Closing this PR as it has been inactive for 10 days.

@github-actions github-actions bot closed this Aug 19, 2025
@dmontagu dmontagu removed the Stale label Aug 19, 2025
@dmontagu dmontagu reopened this Aug 19, 2025
@dmontagu dmontagu mentioned this pull request Aug 19, 2025
Closed
@dmontagu dmontagu force-pushed the dmontagu/graceful-evals-error-handling branch from 060d305 to e801cf3 Compare August 26, 2025 04:43
@hyperlint-ai Hyperlint AI
Copy link
Contributor

hyperlint-ai bot commented Aug 26, 2025

PR Change Summary

Enhanced error handling in evaluations with added retry functionality, impacting report structures.

  • Introduced retry functionality for HTTP requests using the tenacity library.
  • Updated error handling in evaluations to improve resilience against transient failures.
  • Modified ReportCase and EvaluationReport classes, potentially breaking existing deserialization.
  • Added new documentation for retry functionality and usage examples.

Added Files

  • docs/api/retries.md
  • docs/retries.md
How can I customize these reviews?

Check out the Hyperlint AI Reviewer docs for more information on how to customize the review.

If you just want to ignore it on this PR, you can add the hyperlint-ignore label to the PR. Future changes won't trigger a Hyperlint review.

Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add hyperlint-ignore to the PR to ignore the link check for this PR.

@dmontagu dmontagu changed the base branch from main to dmontagu/fix-retry-behavior August 26, 2025 04:44
@dmontagu dmontagu force-pushed the dmontagu/graceful-evals-error-handling branch from 25b903a to cad8eaf Compare August 26, 2025 04:50
Base automatically changed from dmontagu/fix-retry-behavior to main August 26, 2025 09:47
Kludex
Kludex reviewed Aug 27, 2025
View reviewed changes
DouweM
DouweM requested changes Aug 27, 2025
View reviewed changes
pydantic_evals/pydantic_evals/dataset.py
span_id=span_id,
trace_id=trace_id,
)
# TODO(DavidM): Address the following TODOs before V1...
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will you create a followup issue for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#2708

@DouweM DouweM enabled auto-merge (squash) August 29, 2025 21:19
@DouweM DouweM merged commit 9354f08 into main Aug 29, 2025
57 of 59 checks passed
@DouweM DouweM deleted the dmontagu/graceful-evals-error-handling branch August 29, 2025 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Kludex Kludex Kludex left review comments

@DouweM DouweM DouweM approved these changes

Assignees

@dmontagu dmontagu

Labels
awaiting author revision
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Handle errors more gracefully in pydantic-evals
3 participants
@dmontagu @DouweM @Kludex